Inter-Task Effects Induce Bias in Crowdsourcing

Microtask platforms allow researchers to engage participants quickly and inexpensively. Workers on such platforms probably perform many tasks in succession, so we investigate interactions between earlier tasks and later ones, which we call inter-task effects. Existing research investigates many task design factors, such as framing, on the quality of responses, but to our knowledge, does not address inter-task effects. We used a canonical image-labeling task on Amazon Mechanical Turk to measure the impact of intertask and framing effects on the focus and specificity of labels that workers provide. We found that inter-task effects had a much stronger impact than framing, and that workers provided more specific labels when labeling a series of images that were similar to one another.